arXiv:1508.05206vl [math.AP] 21 Aug 2015 


THE BREZIS-NIRENBERG PROBLEM FOR NONLOCAL SYSTEMS 


L.F.O. FARIA, O.H. MIYAGAKI, F.R. PEREIRA, M. SQUASSINA, AND C. ZHANG 

Abstract. By means of variational methods we investigate existence, non-existence as well 
as regularity of weak solutions for a system of nonlocal equations involving the fractional 
laplacian operator and with nonlinearity reaching the critical growth and interacting, in a 
suitable sense, with the spectrum of the operator. 
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1. Introduction and results 


Let n C be a smooth bounded domain. In 1983, Brezis and Nirenberg, in the seminal 
paper [3], showed that the critical growth semi-linear problem 


( 1 . 1 ) 


—Art = Xu + in Q, 

< u > 0 in n, 

u = 0 on dQ, 


admits a solution provided that A € (0, Ai) and Y > 4, Ai being the hrst eigenvalue of —A 
with homogeneous Dirichlet boundary conditions and 2* = 2N/(N — 2) the critical Sobolev 
exponent. Furthermore, in dimension N = 3, the same existence result holds provided that 
^ < A < Ai, for a suitable /r > 0 (if H is a ball, then fi = Ai/4 is sharp). By Pohozaev 
identity, if A ^ (0, Ai) and II is a star-shaped domain, then problem (1.1) admits no solution. 
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Later on, in 1984, Cerami, Fortunato and Struwe obtained in [6] multiplicity results for the 
nontrivial solutions of 


f —Att = Au + tt^* ^ in n, 

\ u = 0 on dQ, 


when A belongs to a left neighborhood of an eigenvalue of —A. In 1985, Capozzi, Fortunato 
and Palmier! proved in [5] the existence of a nontrivial solution of (1.2) for all A > 0 and 
A > 5 or for A > 4 and A different from an eigenvalue of —A. Let s G (0,1) and A > 2s. 
The aim of this paper is to obtain a Brezis-Nirenberg type result for the following fractional 
system 


(1.3) 




(—A)®u = au + bv + 

{—AYv = bu + cv + 
u = v = 0 


2p 

p + q 
2g . 

p + q 






in n, 

in n, 
in \ 


where (—A)* is defined, on smooth functions, by 


(—A)^u(x) := C{N,s) lim f 

e\o Jm 


u{x) - u{y) 


dy, X € 


V.N 


C{N, s) being a suitable positive constant and p,q > 1 are such that p + g is compared to 
2* := 2A/(A — 2s), the fractional critical Sobolev exponent [8]. The corresponding system 
in the local case was studied in [1]. For positive solutions, system (1.3) turns into 

{—AYu = au + bv-\ —in 11, 

P + q 


(1.4) 


(-A)^u 


bu + cv + 


2g 

P + q 




u > 0, V > 0 
u = v = 0 


in Q, 
in 11, 

in \ Q. 


In the following we shall assume that H is a smooth bounded domain of with A > 2s and 
we shall denote by (Aj^s) the sequence of eigenvalues of (—A)^ with homogeneous Dirichlet 
type boundary condition and by and iJ .2 the real eigenvalues of the matrix 


A := 



a,b,cG M. 


Without loss of generality, we will assume pi < P 2 - By solution we shall always mean weak 
solution in the sense specified in Section 2, where the functional space A(I1) is fully described. 
It is known that the first eigenvalue Ai^^ is positive, simple and characterized by 


(1.5) 


\{—A) 2 ufdx 


Ai s = inf 

’ -uGX(fI)\{0} 


/ 


\u\‘^ dx 


The following are the main results of the paper. 
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Theorem 1.1 (Existence I). Assume that b > 0, fj ,2 < and 

p + q<2:. 

Then (1.4) admits a solution. 

Theorem 1.2 (Existence II). Assume that b >0 and 

p + q = 2*. 

Then the following facts hold. 

(1) // > 4s and 0 < then (1.4) admits a solution; 

(2) If 2s < N < 4s, there is fi > 0 such that (1.4) admits a solution ifp. < pi < < Ai,s. 

Theorem 1.3 (Nonexistence). Assume that p + q = 2^ and one of the following facts hold. 

(1) H is star-shaped with respect to the origin and p 2 < 0; 

(2) H is star-shaped with respect to the origin and A is the zero matrix; 

(3) b > 0 and pi > — |a — c|. 

Then (1.4) does not admit any solution. Furthermore, if P 2 ^^ o.nd 

P + q>2*, 

then (1.4) does not admit any bounded solution ifH is star-shaped with respect to the origin. 

Theorem 1.4 (Regularity). Assume that p-\-q < 2*. If {u,v) is a solution to (1.3), then 
u, u € for s E (0,1/2) and u,v £ C'ior(^) ^ ^ (1/2) 1). In particular {u,v) solves 

(1.3) in classical sense. 


The nonexistence result stated in (3) of Theorem 1.3 holds in any bounded domain. For 
6 = 0 it reads as p 2 > Ai,s, properly complementing the assertions of Theorem 1.2. The 
above results provide a full extension of the classical results of Brezis and Nirenberg [3] for 
the local case s = 1. We point out that we adopt in the paper the integral definition of the 
fractional laplacian in a bounded domain and we do not exploit any localization procedure 
based upon the Caffarelli-Silvestre extension [4], as done e.g. in [2]. See [14] for a nice 
comparison between these two different notions of fractional laplacian in bounded domains. 
By choosing p = q = 2*/2, system (1.4) reduces to 


( 1 . 6 ) 


' {-AYu = au + bv + ^2s/(JV-2s)^7V/(iV-2«) 

{-AYv = bu + CV + u^/(^-^Ay2s/iN-2s) 
u > 0, u > 0 in n, 

u=v = 0 in \ n, 


which, in the particular case of a = c, setting u = v, boils down to the scalar equation 


(1.7) 


' {—AYu = \u + ^ in n, 

< u > 0 in n, 

u = 0 in \ n, 


which is the natural fractional counterpart for the classical Brezis-Nirenberg problem [3]. 
For existence results for this problem, we refer to [12,13] and to the references therein. 
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2. Preliminary stuff 


2.1. Notations and setting. We refer the reader to [8] for further details about the func¬ 
tional framework that follows. For any measurable function u : —)• M we define the 

Gagliardo seminorm by setting 


. , (CiN,.,) f 




R2iV |x - 


( f \{—A)2u\‘^dx 


1/2 


The second equality follows by [8, Proposition 3.6] when the above integrals are finite. Then, 
we introduce the fractional Sobolev space 

= {u e : [tijs < oo}, = ( 11 ^ 11^2 + 

which is a Hilbert space and we consider the closed subspace 

(2.1) X{n) := {u G : u = 0 a.e. in R^ \ H}. 

Due to the fractional Sobolev inequality, X(D) is a Hilbert space with inner product 


( 2 . 2 ) 


{u,v)x ■■= 


C{N,s) 


[ 

Jr2JV 


{u{x) - u{y)){v{x) - v{y)) 




- y\ 


dx dy, 


which induces the norm || • ||x = [-js- Now, we consider the Hilbert space given by the 
product 

(2.3) Y{n) := x{n) X x{n), 

equipped with the inner product 

(2.4) {{u, v), ((/?, iP))y ■■= {u, ip)x + (t, 
and the norm 

(2-5) II(^^,^^)I|y := {\\u\\x + IHxY^'^- 

We shall consider L™’(H) x L™’(H) (m > 1) equipped with the standard product norm 

( 2 . 6 ) \\{u,v)\\LmxL^ ■= iWuWlm + \\v\\\mY/‘^. 


We recall that we have 

(2.7) < (HP,[/)r 2 </rsjC/p, for all P := (w, u) G M^. 

In this paper, we consider the following notation for product space 5^ x 5 := and set 

t(;'''(x) := max{rc(x),0}, w~(x) := min{rc(x),0}, 

for positive and negative part of a function w. Consequently we get w = + w~. During 

chains of inequalities, universal constants will be denoted by the same letter C even if their 
numerical value may change from line to line. 
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2.2. Weak solutions. Consider the following system 

{—Ayu = f{u,v) in 0 , 

( 2 . 8 ) ' 


{—Ayv = g{u,v) in 

u=v = 0 in \ n. 
are Caratheodory mappings which satisfies, respectively, the following 


where /, <7 : M x M - 
growths conditions 

(2.9) \f{z,w)\<C{l + \zf^-^ + \w 

( 2 . 10 ) \giz, u))| < 0(1 + + k 




for all {z, w) G M^, 
for all {z, w) G 


Definition 2.1. We say that {u,v) G Y{Q) is a weak solutions of (2.8), if 

(2.11) {{u,v),{(p,y))Y = / f{u,v)ipdx+ / g{u,v)ydx, 

Ju JQ 

for all {ip,y) G y(n). 

2.3. A priori bounds. We introduce some notation: for all t G M and k > 0, we set 

(2.12) tk := sgn(t) min{|t|. A:}. 

From [9, Lemma 3.1] we recall the following 

Lemma 2.2. For all a, 6 G M, r > 2, and k > 0 we have 

{a - b){a\a\l-^ - blbf-^) > li!-^(a|a||"^ - 6 | 6 ||"V- 

In the following, we prove an L°°-bound on the weak solutions of (2.8) which will be needed 
in order to get nonexistence and regularity results. 

Lemma 2.3. Assume that f and g satisfy (2.9)-(2.10) and let {u,v) G Y{Q,) be a weak 
solution to (2.8). Then we have u, u G L°°(n). 

Proof. For all r > 2 and A; > 0, the map 1 1 —is Lipschitz in M. Then 

{u\u\l-yO) G y( 0 ), (0,u|u|^-2) G y( 0 ). 

We test equation (2.11) with (rt|tt|^~^, 0), we apply the fractional Sobolev inequality, Young’s 
inequality, Lemma 2.2, and use (2.9) to end up with 

n , T-1,,9 . . Cr^ 


\u\u\ 


1 ^ 2 * < C\\u\u\l ^11^ < ■^^{u,u\u\l ‘^)x 


<Cr \f{u,v)\\u\\u 


r-2 


dx 


(2.13) 


< Cr 


(kll'^lfc ^ + l'“l 


\u 


r-2 


+ U 


i2i-i 


u\\u 


r-2 


) dx 


+ \ufs+r-2 + |^|2‘+r-2) 


< Cr 

for some C > 0 independent of r > 2 and A; > 0. Then, Fatou Lemma, as A: —>■ 00 yields 


(2.14) 


\u\\l.^ 2 , < Cr 




+ Li 


+ T 


dx 
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where 7 = (2*/2)^/^ (the right hand side may at this stage be cx)). Now, in a similar way, 
test ( 2 . 11 ) with ( 0 , to obtain for some C > 0 independent of r > 2 

(2.15) ||i;|| 2 , 2 , < ^ dx ), 

(the right hand side may be 00 ). By (2.14) and (2.15) we get 


(2.16) 




^ < Cr 


\u\-^ + |i;r-^ + + \vfs+r-2^^ dx 


Our aim is to develop a suitable bootstrap argument to prove that u,v G LP(n) for all p > 1. 
We start from (2.16), with r = 2* + 1 > 2, and fix u > 0 such that Cra < Then there 
exists Kq > 0 (depending on u and v) such that 


1_ ±. 

2 ^ 


1_ ±. 

2 ^ 


(2.17) 


/ dx 

\J{\u\>Ko} J 

By Holder inequality and (2.17) we have 


+ [ / dx 

\ ./{|i;|>A'o} > 


< a. 


A |y|2*+r 2 ^^ < ^|{|'u| < 77o}| + 

Jn 


\u 


2t+r-2 


dx 


'{\u\>Ko} 


(2.18) 

and 

(2.19) 


< ( j {u^)^dx 

2g H-r— 21 pj I , II II r 

1^1 +^ll^ll^^ 2 . 




{\u\>Ko} 


<^0 


/ 

Jn 


|2t+r-2 


dx < ^|H| + (T||ur^ 2 ,. 


By (2.16), (2.18) and (2.19), we have 


( 2 . 20 ) 


2 + \\V\\j^^2 


' < Cr 


+ |u|"' dx + 


2j(2j+l) 

Since r = 2* + 1, we get u,v G L 2 (fl). We define a sequence {r„} with 


Since 


ro = 2 : + 1 , rn+i = Yrn -2^+2. 

o* , ,^ 2 ,*( 2 ,* + l) 

2 ^ + ro - 2 < ---, 


we get 

II'*^IIl 2 S+'- 0-2 + lbllL 2 S+’- 0-2 < + 00 . 

Hence, we aim to begin an iteration in order to get the L°°-bounds of u and v. Using formula 
(2.16) and Holder inequality, we obtain 




< (C'r)F(|ll| 2 J+^(||rt 


ll2^..-2 + II-IIS.. 


- 2 ) + 


tt 


|2*+r-2 




|2*+r-2 


L2%+t- 2 ' \\'^\\i^2l+r-2 

r-l . /u II , M M ^2*+r-2 


< (Cr)-- ( (1 + |H| 2 | ) (||u||j;^ 25 +r -2 + ||u||j;^ 2 |+r- 2 ) + (||t6|| j^ 2 |+r -2 + Ill'll J^ 2 |+r- 2 ) 


1 
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Substituting r^+i for r, since = 2* + r^+i — 2, we get 

( 2 . 21 ) 

< (C't-^+i)’■"+! ^C(||n||^^2^„ + ||u||^^2^^) + ( 

Denote 

Tn ■= max{l, ||n||^^ 2 ,„ + ||u||^^ 2 ,„}. 
Then (2.21) can be written as 




'^'r'^rn'j r^+1 


(2.22) Tn+1 < (l + C)’-"+ir„+i’-"+ir„’'-+i. 

Since r„+i = — 2* + 2, by induction it is possible to prove that 

rn+i = 2 * - 1 + 27 - 2 ’"-^ nGN. 


v2n+2 s 


If n = 0 the assertion follows by a direct calculation. Assume now that the assertion holds 
for a given n > 1 and let’s prove it for n + 1. We get 

rn+2 rn+i 2 * - 2 


^271+4 ^271+2 ry2Tl+4 


= 2: - 1 + 27 

= 2 : - 1 + 27 


-2n-2 2* — 2 


7 


.2n+4 


-2n-4 


which proves the claim. In particular, ~ 2* — 1. Prom (2.22), we also have 


Tn+i < (l + C)’'"+ir„+r-+ir„’'-+i 

1 1 / J_ J_ \ ’’n + l 

< (1 + C')’'"+lrn+i'’’^+l (^(1 + C')’'"rn’-"r„_i j 


1 +T^ 


= (1 + C')’'"+ir„+i'’"+ir„"’"+ir„_i '■"+1 <••• 


< (1 + C) ^"+1 I '■'*+! r„’'"+i r„_i’’"+1 • • • ri’’"+1 iTq ’''‘+1 




= (l + C)(-^-i)’'"+i 

i=0 


2(71 — 1 ) 


^n +1 


Tn "-+1 . 


We can easily compute that 

^ 2 n +2 _ -y 


2 * + 1 


( 72 -l)r„+i (2 *-l)(2*-2)’ 

Moreover, rj+i < for every i G N, since 


Tn+l 


2*-l 


D+l 


21-2 r, 


..^ 2 j +2 ..^ 2 i ..^ 2 i +2 < ^2i <■■■ <f0, 
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and rn +1 > 7^”+^ eventually for n large since ^ 7 ^ 2* — 1 > 1, so that 

n in 

i=0 i=0 

Hence (T„) remains uniformly bounded and the assertion follows. Notice that the L°°-bound 
depends on Tq which depends on u (and not only on ||u||2j) through the presence of Kq > 0 
in estimate (2.20). □ 


y2n + '2 


2 ^ 1 


7' 


i=0 T 


■M+2 


< +00. 


3. POHOZAEV IDENTITY AND NONEXISTENCE 


The purpose of this section is to prove Theorem 1.3, for this we need the following auxiliary 
result known as Pohozaev identity for systems involving the Laplacian fractional operator. 


Lemma 3.1. Let Ll be a bounded domain and let F € C^(M'’“ x M"*") be such that 
and Fy satisfy the growth conditions (2.9) and (2.10). Let {u,v) G P(H) be a solution to 
system 


(3.1) 


'{—Ayu = Fu{u,v) inLl, 

< {—Ayv = F^{u,v) inLl, 

u=v = 0 in'R^ \Q. 


Then n, u € C'*(M'^), G (^^^“(fl) for s G (0,1/2), G (^(^’“(fl) for s G (1/2,1) and 
(3.2) ^ /of some a G (0,1), 

where 5{x) := dist(x,5H), meaning that u/(5®|o and u/(5^|f2 admit a continuous extension to 
Q which is C°‘{Ll). Moreover, the following identity holds 


(3.3) 


[ ("|(—A)2n|^ + |(—A)2tip'] dx — 2* f F{u,v)da 

Jrn V ' Jn 

r(i + s)2 f 


+ 


N -2s 

where P is the Gamma function. 


idn 


L \S‘ 


TT + TT 


{x,u)^Nda = 0 , 


Proof. In light of Lemma 2.3, we learn that u, u G L°°{Ll). Then, Fu{u,v) and Fy{u,v) 
belong to L°°{Ll) too. In turn, by [10, Theorem 1.2 and Corollary 1.6], we have that u and 
V satisfy the regularity conclusions stated in (3.2). In particular, the system is satisfied in 
classical sense. Whence, we are allowed to apply [11, Proposition 1.6] to both components 
u and V, obtaining 


/ {x ■ Vu){—Ayudx = 

Jn 

f {x ■ Vv){—Ayv dx = 

Jn 


2s-N 

2s-N 


u{—Ayudx — 


r(i + s) 


'an 


[ v{-A) 
Jn 


—Ayv dx — 


r(i + s)2 


/ 

Jan 


{x,iy)^Nda, 


(x, n)^Nda. 


Then, since (—A)^u = Fu{u,v) and {—A)^v = Fy{u,v) weakly in H and recalling that 
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we get 


(x • Vu)Fu{u, v) dx = 
(x ■ Vv)Fy{u, v) dx = 


2s-N 
2 

2s-N 


f |(-A)^/\pdx -f {x,iy)^Nda, 

JRN 2 Jqq VdV 

Observing that VF{u, v) ■ x = Fu{u, v)S/u ■ x + Fv{u, v)Vv ■ x, integrating by parts we get, 

{2s — N) f (\{—A)iu\‘^ + \{—A)iv\‘^')dx + 2N f F{u,v)dx 
JRN \ ' Jn 


which concludes the proof. 


□ 


3.1. Proof of nonexistence. Consider hrst the case p + g = 2* with assumption (1) and 
assume by contradiction that (1.4) admits a positive solution {u,v) G y(Q). Consider the 
functions /, g : M"'' x M+ —>■ M dehned by 


f{z, w) = az + bw + 


2p 

p + q 


■zP-^w\ 


g{z, w) = bz + cw + 


2g 

P + q 


■zPw'i-\ 


Then, setting 


F{z, w) 


-z'^ + bzw + + -^zPw'i 

2 2 P + q 


1 


iA{z,w), {z,w) 


12 + -—zPw'^, 
p + q 


we obtain that F G x M^), Fz = f and Fw = g satisfy the growth conditions (2.9) and 

(2.10) and (u, u) is a weak solution to (2.8). Then, the components u,v enjoy the regularity 
(3.2) stated in Lemma 3.1 and identity (3.3) holds. Testing (2.11) with (</?, V’) = ('w,u), yields 



^|(—A)2up + |(—A)2xp^ dx 


f{u,v)udx+ / g{u,v)udx 

i Jn 

{AU, U)^2dx + “2^ f vFv^dx^ 
i Jn 


which substituted in (3.3), yields, recalling that p + g = 2*, 


1 - 


{AU, U)-^2dx + 


r{i + sf 
N-2s 


I 

Jan 


+ 


2i 


(x, i')^Nda = 0. 


Since fl is star-shaped w.r.t. the origin, the equation above yields f^{AU, U)^ 2 dx > 0. This 
is a contradiction with (2.7), because /U2 < 0 and u, u > 0. Now we cover case (2). If A is 
the zero matrix, we get 



which contradicts the fractional version of Hopf lemma, see [9, Lemma 2.7], since (—A)*m > 0 
and {—AYv > 0 weakly yield ^ > OJ and ^ > uj', for some positive constants uj,uj'. Let us 
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turn to case (3). If (/?! > 0 is the first eigenfunction corresponding to and we assume 
that a solution of (1.4) exists, by choosing (¥?i,0) and (0,(^i) respectively in (2.11), we get 

Xi s [ uifidx = I {—A)^u{—A)^(pidx= [ (auipi + bvipi - —u^~^v'^(pi)dx, 

’ Jn Jrn p + q 

Xi s I vipidx = [ (—A)^v(—A)^(pidx= [ (buipi + cvipi - —u^v'^~^(pi)dx. 

’ Jn Jrn p + q 

Then, since b >0 and u,v > 0, we get 

/ uifidx > a u+idx, Ai^^ / vipidx >c v+idx, 

Jn Jn Jn Jn 

that is max{a, c} < Ai^g. On the other hand, by assumption and a direct calculation 


I I ^ (a + c) - i/(a - c)2 + 462 (-q g) _ |q _ g| 

Ai,s -\a-c\< pi = - ^ -< - - -= mm{o, c}, 

which yields max{a, c} > Ai^^, namely a contradiction. Finally we prove the last assertion. In 
the case p + q > 2*, any bounded solution of system (1.4) is smooth according to Lemma 3.1 
and arguing as above yields the identity 


(l-f) / {AU,U)^.dx+ 2(1-^) [ vFv<idx 
^ ^ ' Jn ^ P + q' Jn 

, r(i + s )2 r 

Jdi 


6 ^) U* 


2 i 


(x, h')^Nda = 0. 


^ “ 2s JQQ 

This yields f^(Al7, U)^ 2 dx > 0, contradicting ^2 < 0 via (2.7). This concludes the proof. □ 
Proof of Theorem 1.4. The assertion follows as a particular case of Lemma 3.1. □ 

4. Existence I, subcritical case 

In this section, we will prove the Theorem 1.1 which guarantees the existence of solutions 
for the problem 1.4 involving subcritical non-linearity. 

4.1. Proof of existence I. Let 11 be a bounded domain and suppose that 

(4.1) 6 > 0, 

(4.2) p2 < Ai,s, 


(4.3) 

Consider the functional I : Y (H) 


p + q <2*. 
defined by 


I{U) ■.= \\\U\\l - \ j^{AU,U)^2dx. 


We shall minimize the functional I restricted to the set 

:= ( E = (n, u) G y(ll) : [ (ir+)P(u+)«dx = 1 

I Jn 

By virtue of (4.2) the embedding Af(ll) L'^{Q,) (with the sharp constant Ai^*), we have 

(4.4) m > ^ min |l, (l “ } WWy > 0- 











So define 
(4.5) 
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Iq := inf /, 

M 

and let {Un) = iun,Vn) C A4 be a minimizing sequence for Jq. Then I{Un) = Io + On{l) < C, 
for some C > 0 (where 0^(1) ^ 0, as n —>■ oo) and consequently by (4.4), we get 

(4:.6) ['^n]s + = ll'^nllx T Il'^riHx ~ ll^n||y ^ C'■ 


Hence, there are two subsequences of {un) C X(H) and {vn) C X(H) (that we will still label 


as Un and Vn) 

such that Un = (un,Vn) Converges to some U = (u,v) in 

(4.7) 

[u]l < lim inf 

/ 

\Unix) - Uniy)]"^ 

(4.8) 

[i;]2 < lim inf 

n 2 

/ 

\Vnix) - lln(y)P , , 

l, “-“y- 


Furthermore, in view of the compact embedding X(H) for all a < 2* (cf. [ 8 , 

Corollary 7.2]), we get that Un = {un,Vn) converges to {u,v) strongly in (LP+'J(H))^, as 
n —>■ oo. Of course, up to a further subsequence, we have that {un{x),Vn{x)) converges 
to {u{x),v{x)) for a.e. x G M'^. Now we will show that U := {u,v) G A4. Indeed, since 
(Un) C A4, we have 


(4.9) 


(Un)^(v;l)‘^dx = 1 . 


Since 

lim f {unl^^^^dx = [ {ul^^'^dx, lim f \vn\^^'^dx = f \v\^~^‘^dx, 

^ Jn Jn ”■ Jn Jn 

we have in particular < ??i and < h 2 ) for some rji G L^(H) and any n G N. 

Then 

(u+)^(x)(i;+)^(x) < ^ \un(x)\^^‘^ H-^|i;„(x)P+'^ < 'qi{x) + mix), for a.e. in H. 

p + q p + q 

In turn, by the Dominated Convergence Theorem, passing to the limit in (4.9), we obtain 


/ {u^)^(v^)'^dx = 1 , 

Jn 

and, consequently U = {u,v) G M. with n, u 7 ^ 0. We now show that U = (u, v) is, indeed, 
a minimizer for I on Ai and both the components u, v are nonnegative. By passing to the 
limit in I(Un) = Iq + 0^(1), where o,i(l) —>■ 0 as n —>■ 00 , using (4.7) and (4.8) and the 
strong convergence of (un,Vn) to (u, u) in (L^(D))^, as n —>■ 00 , we conclude that /(D) < Iq. 
Moreover, since U G M. and Iq = infx / < I(U), we achieve that I{U) = Iq. This proves 
the minimality of 17 G M. On the other hand, let 

G(U) = [ (u+)P(u+)'?dx - 1, 

Jn 

where U = {u,v) G Y(Il). Note that G G and since U G M, 


G'(U)U={p + q) [ (u^)^(v^)^dx = p + q ^ Q, 
Jn 
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hence, by Lagrange Multiplier Theorem, there exists a multiplier /r E M such that 
(4.10) i:) = ^), V((^, lA) e Y{n). 

Taking (v?,i/') = {u~,v~) := U~ in (4.10), we get 


\\U 


-I|2 

\y 


CiN,s) 

2 

C{N,s) 


I 

Jk 

f 

Jr 


u'^{x)u (y) + u {x)u'^{y) 

R2iv |x - y\N+2s 

v'^ {x)v~ (y) + v~ {x)v~^ (y) 


_ ^i\N-\-‘2s 


\x - y\ 


dxdy 


dxdy 


[ {AU,U-)^2dx. 
Jn 


Dropping this formula into the expression of I{U ), we have 


(4.11) I{U ) = - / {v~^u +u'^v )dx + 


C{N,s) f u+(x)n {y) + u (x)n+(y) 


+ 


2 

C{N,s) 


[ 


/ 

Jr2JV 


\x — 7/|^+2« 


dxdy 


v~^{x)v (y) + V {x)v~^{y) 


\x-y\ 


N+2s 


dxdy < 0, 


since b > 0, w <0 and w~^ > 0. Furthermore, 


/([/-) > min |l, I ||t^”||y > 0 

and using (4.11), we get U~ = {u~,v~) = (0,0) and therefore u,i; > 0. We now prove the 
existence of a positive solution to (1.3). Using again (4.10), we see that 

||?7||y — f {AU,U)^ 2 dx — ii{p + q) f vPv'^dx = f) 

Jn Jn 

and since U G A4, we conclude that 

= /([,) = dp±L > 0 . 


since Iq is positive, via (4.2). Then, by (4.10), U satisfies the following system, weakly, 


{—AYu = au + bv + vP 


p + q 

{—AYv = bu + cv + vPv^~^ 


u = V = 0, 


p + q 


n 

R^\n. 


Now using the homogeneity of system, we get r > 0 such that W = {loYU is a solution of 
(1.4). Since 6 > 0 and u,v > 0 we get, in weak sense 

(—A)®u > au Q 

(—A)®u > cv D 

u > 0, V > 0 Q 

_ U = v = 0, R^\n. 

By the strong maximum principle (cf. [9, Theorem 2.5]), we conclude u, u > 0 in D. □ 
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5. Existence II, critical case 

Next we turn to Theorem 1.2, for the critical case p + g = 2*. The variational tool used is the 
Mountain Pass Theorem. The embedding X(II) (II) is not compact, but we will show 

that, below a certain level c, the associated functional satisfies the Palais-Smale condition. 


5.1. Preliminary results. We will make use of the following definition 


(5.1) 
where 

(5.2) 


Ss := inf Ss{u), 
nGA:(n)\{o} 


|(—A)2nfdx 


S.Ju) := 


\u{x)\‘^‘‘dxj 

is the associated Rayleigh quotient. We also define the following related minimizing problems 

f \{—A)2u\‘^dx 


(5.3) 


and 


(5.4) 


Sp+q{Q) := inf 

«ex(n)\{0} 


2 

, p+q 


<Sp„(n) := inf 

u,v eA(O)\{0} 


( f |u(x)|P+«dxV 

^|( —A) 2 Mp + |( —A) 2 T;p^ dx 


\u{x)\^\v{x 


2 

p+q 


We shall also agree that 

Ss = Sp+q{Q), Ss := Sp^q{Q), iip + q = 2*. 

The following result, in the local case, was proved in [1]. The proof follows by arguing as it 
was made in [1], but, for the sake of completeness, we present its proof. 


Lemma 5.1. Let LI be a domain, not necessarily bounded, and p + q < 2*. Then 


(5.5) 


Sp,q{Ll) 



Sp+q(Ll). 


Moreover, if wq realizes Sp+q{Ll) then {Bwo,Cwq) realizes Sp^q{Ll), for all positive constants 
B and C such that BjC = sjpjq. 


Proof. Let {wn} C A(II) \ {0} be a minimizing sequence for Sp+q{Q). Define Un ■= swn and 
Vn := tWn, where s,t > 0 will be chosen later on. By definition (5.4), we get 

g(^) [ \{-Af^Wn\^dx 

(5.6) ^ > Sp^qiLl), 

( / \wn{x)\P+‘^dx)^^‘’ 

where g : is defined by 

g{x) := + X-2P/P+'', X > 0. 
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The minimum value is assumed by g at the point x = ^/pjq, and it is given by 


9{VWq) = 


P \ P+9 I /P\ P+1 


+ (- 


Whence, by choosing s,t in (5.6) so that s/t = y/p/q, and passing to the limit, we obtain 

Sp,gi^) < gi^/I^)Sp+g{n). 

In order to prove the reverse inequality, let {{un,Vn)} C (-T(n) \ {0})^ be a minimizing 
sequence for Sp^q{Q) and define Zn := SnVn for some > 0 such that 

[ = f \Zn\^'^'^dx. 

Then, by Young’s inequality, we obtain 

[ \Unf\Zn\'^dx < ^ [ \Un\^~^'‘dx-\ - — [ \Zn\^~^'^dx 

Jr^ p + q p + q Jr^ 

(5.7) = f = f |z„|P+'?dx. 

Jrn Jrn 

Thus, using (5.7), we obtain 

f (\{-A)iun\‘^+ \{-A)ivnf) dx 
Jrn \^ 

( [ \Un{x)\P\Vn{x)\‘^dx)'’^'‘ 

^Jrn ^ 

(^|(-A)tu„p + |(-A)iunp^ dx 


_ „2qlv+q jRN 

- 


> s 2 <?/P+<?. 


RN 


\Un{x)\P\Zn{x 


2 

P+q 


[ l(-A) 

Jrn 


—A)2ti rdx 


2 

1 P+1 


+ s 


-2plp+g_ 


[ l(-A) 

Jrn 


— A)2Zri\‘^dx 


( [ lUnixW+'^dx) 

^Jrn > 

— 5('®n)‘5p+g(n) > g{^\Jpjq)Sp-i^q{id). 


.(x)r+'?dx) 


2 

p+(l 


Therefore, letting n —>■ oo in the above inequality, we get the reverse inequality, as desired. 
From (5.5), the last assertion immediately follows and the proof is concluded. □ 

From [7, Theorem 1.1], we learn that Ss is attained. Precisely Sg = Ss{u), where 

^ , __^ / \ ^ 

(5.8) u{x) = 


, X € M^, k eR \ {0}, p > 0, xo € 


(/i^ + |x — xop) 2 


Equivalently, 


Ss = inf 

u e A(n) \ { 0 } 


—A) 2 up(ix, 


oN 


= 1 


f \{— A) 2 dx = f |(—A) 

Jrn Jrn 
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where u{x) = i/(x)/||tt||^ 2 S • In what follows, we suppose that, up to a translation, xq = 0 in 

(5.8). The function 


u*{x) := m(^), 


X G 




is a solution to the problem 

(5.9) 

verifying the property 

(5.10) 

Define the family of functions 


(-A)*?i = in 


Il2s 


Ue{x) = e — —u* 


X G 




then Ue is a solution of (5.9) and verifies, for all e > 0, 


(5.11) 


[ \i-A)me\^dx= [ \Ueix)f^dx = S^/^\ 


Fix d > 0 such that B^s C D and rj G a cut-off function such that 0 < ry < 1 in 

T] = 1 in Bs and 77 = 0 in B^^ = \ B25, where B^ = Br{0) is the ball centered at origin 

with radius r > 0. Now define the family of nonnegative truncated functions 

(5.12) u^ix) = r/(x)C4(x), x G M'^, 

and note that G X(D). The following result was proved in [13] and it constitutes the 
natural fractional counterpart of those proved for the local case in [3]. 

Proposition 5.2. Let s G (0,1) and N > 2s. Then the following facts hold. 

\{-A)^Ue\‘^dx < + C)(e^-2.), as e ^ 0. 


[ 

JR^ 


b) 


[ 

JR^ 


if 

U£{x)\"^dx > I loge|-|-©(e^^) if 


CsE^^ + C>(e"-2") 


N > 4s, 
N = As, 


c) 


CsE^-^^ + 0 {e^^) if 2s<N < 4s, 
os e —>■ 0. Here Cg is a positive constant depending only on s. 

[ \u£{x)\‘^*^dx = + 0 {e^), as e—>■ 0 . 

Jr^ 


Consider now, for any A > 0, the following minimization problem 


•Ss.A := inf 

veX(Q)\{o} 


where 


<Ss,\(v) := 


f \{—A)2vfdx — X f |u(x)|' 

Jrn Jrn 


dx 


RN 


\v{x)\'^‘‘dx 
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The following result was proved in [13, Propositions 21 and 22] for the first assertion, and 
in [12, Corollary 8] for the second assertion. 

Proposition 5.3. Let s € (0,1) and N > 2s. Then the following facts hold. 

a) For N > 4s, 

<Ss,x{ue) < <5s, for all X > 0 and any e > 0 sufficiently small. 

b) For 2s < N < 4s, there exists > 0 such that 

Ss,\{us) < Ss, for all A > A* and any e > 0 sufficiently small. 

Proof. For the sake of the completeness, we sketch the proof. 

Case: N > 4s. By Proposition 5.2, we infer 

< sr‘ + 0(e--^-)-XC.e^- 

< Ss + - XCse^f 

<Ss + e^^{0{e^-^^)-XCs) 

< Ss, for all A > 0 and e > 0 small enough and some Cs > 0. 

Case: N = 4s. 

c ... 1 logej + 

^s,X\'^e) _ 2 

+ 0(e^)) 

<>S, + 0(e2^)-AC,e2-|loge|, 

<5, + e2^(0(l)-AC,|loge|) 

< 5s, for all A > 0 and e > 0 small enough and some Cs > 0. 

Case: 2 s < N < 4s. 


^ + 0(e^-2^) - XCsE^-^^ + 0{e^^) 

Ss,x{Ue) < - -Y - 

< 5s + e^-2^(0(l) - XCs) + 0{e^^), 

< Ss, for all A > 0 large enough (A > As), s sufficiently small and some Cs > 0. 

This concludes the sketch. □ 


Even if it is not strictly necessary for the proof of our main result, we state the following 
Corollary for possible future usage. 


Corollary 5.4. Suppose that fii given in (2.7) is positive and let 


Ss A = inf Ssa(u,v), 
’ u,v GA(0)\{0} ’ ^ ^ 
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where 


[ (|(-A)fn|2 + |(-A)fi;p) 

\ dx — J 

f {A{u{x),v{x)),{u{x),v{x)))^ 2 dx 

RJV 

1 

(j 

f 

|u(x)|P|' 

v{x)\‘^dx'^ 

2 


Ss,a{u,v) = 


where p + q = 2*. Then the following facts holds 

a) If N > 4s, then 

Ss,A < Ss- 

b) For 2s < N < 4s, there exists > 0, such that if pi > pi^s, we have 

Ss,A < Ss- 

Proof. Prom Proposition 5.3, we have 

a) For > 4s, we have 

'Ss,^iiue) < Sg, if pi > 0 and provided e > 0 is sufficiently small. 

b) For 2s < N < 4s, there exists pi^g > 0, such that if pi > we have 

Ss,^ii{ue) < Sg, provided e > 0 is sufficiently small. 

Let 5,(7 > 0 be such that ^ From (2.7) and the above inequalities, we infer that 

Sg^A ^ Cugf) 


< 


{B^ + C^)\ 

ij 

1 \{-A)2Ue\‘^dx - Pi 1 

R^ Jl 

\us{x)\‘^dx) 

gJV / 

{Bpciy/‘^s 1 

ij 

1 \Ug:{x)\‘^‘>dx 

RN 

^2/2* 


< 


l\q 

(p 

Xq 


p\q/p+q ^ fp\-p/p+<t 


p^q/p+q ^ fp'^-p/p+q 




— Sg. 


This concludes the proof. 


□ 


5.2. Proof of existence II. In order to get weak solutions to system (1.4), we now define 
the functional J : Y (fl) —R by setting 

J{u,v) = ]- [ (\{-A)iu\‘^ + \{-A)2 v\‘^') dx 

2 Jrjv V / 

[ {Mu,v),{u,v))Tg2dx -[ {u'^Yiv^ydx, 

2 Jrn Ig J^N 


whose Gateaux derivative is given by 
(5.13) J'{u,v){ip,y) 

C{N,s) r {u{x) - u{y)){ip{x) - (p{y)) + {v{x) - v{y)){ilj{x) - ^{y)) 

2 jR2iV 


dxdy 


|x — y|^+2« 

{A{u,v),{p,'if))^ 2 dxf {u'^)P-^{v^y(pdx f (v^y-^iu'^yydx 

Jn Jn 
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for every ((/?, ij)) (O). We shall observe that the weak solutions of problem (1.4) correspond 
to the critical points of the functional J. Under hypothesis 0 < our goal is 

to prove Theorem 1.2. We first show that J satisfies the Mountain Pass Geometry. 


Proposition 5 . 5 . Suppose /i2 < The functional J satisfies 

a) There exist I 3 ,p > 0 such that J{u,v) > fi if ||(rt, ^)IIy = p; 

b) there exists (01,62) € y(U)\{( 0 , 0 )} with ||(ei,e2)||y > p such that J(ei,e2) < 0 . 

Proof, a) By means of (2.7), using 

(u+)P(u+)‘' < \u\P+‘> + \v\P+'i = \uf^ + 
and Poincare inequality, we have 

J{u,v) > ^(1 - ^)||(n,i;)|||- - C'||(«,i;)||y% 
where C > 0 is a constant. 

b) Choose {uo,vo) E y(U) \ {(0,0)} with uq > 0, vq > 0 a.e. and uqVq 0. Then 

J{tuo,tvo) = \- [ (|(-A)5mo| 2 + |(_A)f{;op) dx 

2 J^N V / 

^2 r r 

-17/ {Muo,vo),iuo,vo))dx -iilvldx, 

4 jRiV Ig J^N 

by choosing t > 0 sufficiently large, the assertion follows. This concludes the proof. □ 


Therefore, by the previous facts, by the Mountain Pass Theorem it follows that there exists 
a sequence {(un,Vn)} C y(U), so called (PS)c-Palais Smale sequence at level c , such that 

(5.14) J{Un,Vn)^C, II J'(rin,fn)|| ^ 0, 

where c is given by 

c= inf max J(^(t)), 

7GriG[0,l] 


with 


T = {7 E C([0,1],y(U)) : 7(0) = (0,0) and J(7(l)) < 0}. 

Next we turn to the boundedness of {{un,Vn)} in y(U). 

Lemma 5.6 (Boundedness). The (PS) sequence {(un,Vn)} C y(U) is bounded. 


Proof. We have for every n E N 

C* T C^ll (lln; Hn) ||y P J(Ufii Vn) 777 (^m Vn)(Un, Vn) 


— (2 2*) (2 2* ) 


> (i _ 1 

.2 2t 


1 - 


P2 




[Un,n 


Tt) '^nllly. 


Since p 2 < the assertion follows. □ 

The next result is useful to get nonnegative solutions as weak limits of Palais-Smale se¬ 
quences. The same argument shows that a critical point of J corresponds to a nonnegative 
solution to (1.3). 
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Lemma 5.7. Assume that b >0 and fi 2 < Ai^s- Let {{un,Vn)} C Y{Q,) be a Palais-Smale 
sequence for the functional J. Then 

lim||(?i-,i;-)||y = 0 . 

n 

In particular, the weak limit [u, v) of the PS-sequence {{un, Vn)} has nonnegative components. 

Proof. By choosing ip := u~ € X(0) and if := v~ G X(0) as test functions in (5.13) and 
using the elementary inequality 

(a — b){a~ — b~) > (a~ — b~)^, for all a,b G M, 

we obtain 

(u(x) - u(y))(u~(x) - u~(y)) + (v(x) - v(y))(v~(x) - v~(y}) 


I 

Jr 


> 


/ 


\x — 

iu-{x) - u-{y)f + (u"(x) - v-{y)f 


-dxdy 


dxdy. 


(u ,v ))iii 2 dx. 


/R2iv |x - y|^+2^ 

Now, note that, since b > 0 and w~ < 0 and > 0, it holds 

/ iMu,v),iu~ ,v~))K 2 dx < 

Jrn Jrn 

In fact, it follows 

{A{u,v),{u~,v~))^2 = {A{u~,v~),{u~,v~))^2 +b{{v+)u~ + (u+)u"), 
< {A{u~,v~), {u~,v~))^ 2 . 

In turn, from the formula for J'{u,v){u~,v~), it follows that 


J'{u, v){u ,v ) > 


C{N,s) f (u {x)-u {y)f + {v {x)-v (y))^ 


/ 

Jr 2 ^ 


\x — y|'^+2^* 

/ {A{u~,v~), {u~,v~))^ 2 dx > I{u~) + I{v~), 
Jn 


dxdy 


where we have set 

C{N,s) f {w{x) — w{y))‘^ 


C{N,s) f {w{x)-w{y)) /" I i 2 j r i2 n ii 2 

Hw) :=- ^————^^^dxdy - /i2 / dx = H, - 1^2 1 | 11^11^2(^2). 

J IK I I J 


On the other hand, by the definition of Ai^^, we have 

I{w) > (l - 

which finally yields the inequality 

J'{u,v){u-,v-) > (l - 

Since {{un,Vn)} C L(II) is a Palais-Smale sequence, we get J'(un, Vn){u~, v~) = 0^(1), from 
which that assertion immediately follows. □ 


From the boundedness of Palais-Smale sequences (see Lemma 5.6) and compact embedding 
theorems, passing to a subsequence if necessary, there exists {uo,vo) G Y{I}) which, by 
Lemma 5.7, satisfies uo,vq > 0, such that {un,Vn) {uo,vo) weakly in L(II) as n —> oo, 
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{un,Vn) —>■ (mO)^^o) Si.e. ill O and strongly in L^{Q) for 1 < r < 2*. Recalling that the 
sequences 

wi:= wl:= {v:^y-\utT, p + q = 2*, 

are uniformly bounded in (Q) and converges pointwisely to lUg = and Wq = 

Vq respectively, we obtain 

^ (wIjWq), weakly in (0), as n —>■ oo. 


Hence, passing to the limit in 

j'{Un,Vn)i(p,llj) = Onil), V (if , l/j) G Y (Q.), aS U ^ OO, 

we infer that (no,no) is a nonnegative weak solution. Now, to conclude the proof, it is 
sufficient to prove that the solution is nontrivial. 

Claim: {uq,vq) ^ (0,0). Notice that if (uo,no) is a solution of system with uq = 0, then 
no = 0. The same holds for the reversed situation. In fact, suppose uq = 0. Then, if 6 > 0, it 
follows no = 0. If, instead, 6 = 0, then c G {^i,/r 2 } < Ai^^. Since no is a solution of equation 

(—A)*no = cno in H, no = 0 on \ H, 


we have that no = 0. Therefore, we may suppose that (no, no) = (0,0). Define, as in [3], 

L := lim [ (|(-A)tn„p + |(-A)tnnp') da;, 

n^co J^N \ / 


L 

2 ' 


c = 


from J'{Un,Vn){Un,Vn) = o„(l), we get 

lim f (n+)P(n+)'?dx 
n^oo J^N 

Recalling that J{un,Vn) = c + 0^(1), thus 
(5.15) 

From the definition of (5.4), we have 

J^ (^|(-A)fn„p + |(-A)fn„pj dx>Ss( j \un{x)\^\vn{x)\^dx 

and passing to the limit the inequality above, we get 

'L\ 2 / 2 * 


sL 

'n' 


L > Sg 

Now, combining this estimate with (5.15), it follows that 
(5.16) 




N\2 


Take R, C > 0 with B jC = -s/p/q and let n^ > 0 as in Proposition 5.2. Fix e > 0 sufficiently 
small that Proposition 5.3 holds and define Ve '■= 'M£/||'ite|li 2 j. Using the definition of Ss,x{u), 
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for every t > 0, we obtain 
J{tBVe,tCVe) 


< 


( / \{-A)2Ve\‘^dx - m / \Ve\‘^dx 


2 t^*‘‘BPC^ 


t\B^ + C^) 


i^e) 


2 t^*‘BPC‘i 


:= V’(O) t > 0 . 


Thus, an elementary calculation yields 


V'max = maxi/' = 


2 s r (^2 + C^) 


By Lemma 5.1 and Proposition 5.3, we conclude that, for e > 0 small. 


N/2s 


V’max < 


2 s f (B^ + C 2 ) ^ 
N \2(BpC'?)2/2S^ 


2 s r 1 

N 1 2 


p\<i/p+<i ^ /p\-p/p+<} 


V 


N/2s 


N\2 J 

Let now 7 € (^([0,1],y(n)) be defined by 

7 (t) := {TtBve,TtCve), t G [0,1], 

where r > 0 is sufficiently large that J{TBve,TCve) < 0. Hence, 7 € L and we conclude that 

2 s /<Sa^^/ 2 s 

J [tUV^^tL^V^) 7: l/'max ^ 

t>0 


2 s /iS, \ J 

C < sup J(7(t)) < sup J {tBVe, tCVe) < V'max < ( TT ) 

te[0,i] *>n \ 2 / 


which contradicts (5.16). Hence (uo,uo) / (0,0) and the proof is complete. Finally, that 
uo > 0 and uo > 0 follows as in the sub-critical case. □ 
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